High available method for border gateway protocol version 4

ABSTRACT

High availability BGP4 is based on redundant hardware as well as redundant software that replicates the RUN state of BGP4. There are two copies, respectively active and backup, of BGP4 running on two separate redundant hardware platforms. All BGP4 internal implementations apply various methods to replicate the running state of BGP4 independently of peer network routers. When this hardware or software fails on one redundant hardware platform, peer routers are unaware of the failure. Internally, based on duplicative states, the local router recovers from the failure and keeps the protocol running. During the recovery period, the local router can bring up a backup again. In the HA architecture, these activities are not detected by peer routers, such that there is no instability to the Internet backbone caused by BGP4 failure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/819,034, filed on Jun. 25, 2007, which is a continuation of U.S.application Ser. No. 10/185,809, filed on Jun. 27, 2002, now U.S. Pat.No. 7,236,453. This application claims priority to all of theseapplications. This application is also related to commonly assigned U.S.application Ser. No. 09/852,223, entitled “SYSTEM AND METHOD FOR TCPCONNECTION PROTECTION SWITCHING,” filed May 9, 2001, now U.S. Pat. No.6,853,617; and commonly assigned U.S. application Ser. No. 10/153,500,entitled “HIGHLY AVAILABLE OSPF ROUTING PROTOCOL,” filed May 23, 2002,now U.S. Pat. No. 7,292,535; the disclosures of which are herebyincorporated herein by reference.

TECHNICAL FIELD

The invention relates to network routers, and more particularly tohighly available Border Gateway Protocol (BGP).

BACKGROUND OF THE INVENTION

BGP Version 4 (BGP4) is used today on the Internet backbone as a majorrouting protocol (see for example, Y. Rekhter and T. Li, “A BorderGateway Protocol 4 (BGP-4),” IETF RFC 1774, (1995); RFC1771,http://www.ietf.org/rfc/rfc1771.txt; and J. W. Stewart III, “BGP4:Inter-Domain Routing in the Internet,” Addison-Wesley (1998)). BGP4 usesTCP/IP to connect peer routers. These routers are called BGP speakers.If a pair of routers has established a BGP connection, then they aresaid to be peers to each other. A router can have more than one BGPpeer. BGP peer connection goes through a negotiating session in whichconnecting peers exchange OPEN messages, containing router ID, asnumbers etc. If negotiations are successful, then the peer connection issaid to be established. Routers will send route update messages, whichwill either advertise new prefixes or withdraw previously advertisedprefixes. A prefix contains an IP address and IP mask pair, which definethe reachability of the network represented by the prefix. Normally, aBGP speaker will establish connections to several peer BGP speakers.Hence, a BGP speaker receives (and sends) prefix updates from/to thesemultiple peers.

A BGP speaker will select its best routes among the received andself-configured routes. The selection procedure can be simple orcomplex, depending on the router route-selection-policy configuration.The best routes will be used for data forwarding of the router. A BGPspeaker sends an update of only its best routes to a peer BGP speaker.

Almost all the Internet traffic is controlled by BGP4, and Internetstability is of great importance. Any disruption to Internet backbonerouting caused by hardware and/or software failure will affectsubstantially all network entities. The stability of backbone routing isheavily dependent upon both hardware and software stability. Theplatforms that run BGP4 software can crash, and any of these failureswill cause instability on the Internet backbone. Other prior artsolutions use redundant hardware. Nevertheless, the peer routers on theInternet backbone still detect peer routers going down and up, whichwill cause instability in the backbone.

BRIEF SUMMARY OF THE INVENTION

The present invention is described herein is a high availability (HA)method for BGP4 that seamlessly hides router failures from Internetpeers.

High availability BGP4, in accordance with embodiments of the presentinvention, is based on redundant hardware as well as redundant softwarethat replicates the RUN state of BGP4. There are two copies,respectively active and backup, of BGP4 running on two separateredundant hardware platforms. All BGP4 internal implementations applyvarious methods to replicate the running state of BGP4 independently ofpeer network routers. When this hardware or software fails, (forexample, BGP4 fails on one redundant hardware platform), peer routersare unaware of the failure. Internally, based on duplicative states, thelocal router recovers from the failure and keeps the protocol running.In the HA architecture, these activities are not detected by peerrouters, such that there is no instability to the Internet backbonecaused by BGP4 failure.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

BRIEF DESCRIPTION OF THE DRAWING

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawing, in which:

FIG. 1 is a block diagram illustrating a hardware overview of a HA-BGP4system, in accordance with embodiments of the present invention;

FIG. 2 is a block diagram illustrating information flow in a HA-BGP4protocol environment; and

FIG. 3 is a block diagram representing the state of HA instancesACTIVE-BGP4 and BACKUP-BGP4 during fail-over and recovery.

DETAILED DESCRIPTION OF THE INVENTION

Dynamic routing protocols are protocols that routers use to communicatewith each other, to decide where the traffic goes on the Internet. In“Highly available (HA) routing protocols”, routing fails over completelyseamlessly. The outside world is unaware that there has been a faultfrom one router to another. The backup software and the backup routertake over seamlessly, such that the outside world is unaware that therehas been a problem. During this recovery process, a central switchfabric in the central data plane of the router continues to forwardtransit traffic in accordance with routing instructions in forwardingtables created and maintained by the control plane.

A number of different architectures build on each other to attain thiseventual goal of ‘high availability.’

The first set of architectures relate to a number of concepts, one ofwhich is TCP socket fail-over (see U.S. application Ser. No. 09/852,223,cited above, the disclosure of which has been incorporated herein byreference). TCP protocol was not designed so that one computer couldquickly take over from another in the event of a TCP failure, but wasdesigned so that one computer could talk to another in a one-to-onerelationship. Border Gateway Protocol (BGP) uses TCP sockets tocommunicate with other routers. Consequently, the groundwork to make BGPhighly available needed a reliable TCP connection that could be takenover seamlessly by another computer without interruption.

BGP exists in an environment of other software, including an operatingsystem which provides services that BGP relies on, for example fail-overdetection and message flow-through. Specifically the operating system isresponsible for synchronizing the active network interface configurationwith the backup. When the backup boots, the operating system on theactive is responsible for making sure that certain interfaces exist onthe backup. Once that is done, routing software is responsible forsynchronizing the static command line configuration from the activedynamic routing protocol (DRP) to the backup DRP. That includes, forexample, setting addresses on the interfaces and certain otherattributes, which are basically configuration items that are manuallyentered at the command line.

In the high availability architecture, BGP4 is run on two redundantphysically separated master control units, such that one is active BGP4instance and the other is backup BGP4 instance. The two BGP4s aresupported by completely separate hardware and OS software linked by anetwork that supports TCP/IP. The active BGP4 instance will be broughtup first, and can establish BGP4 peer connections and exchange routinginformation with the peers, just as a conventional BGP4 instance woulddo. The backup BGP4 instance will be brought up after the activeinstance is up. The backup BGP4 does not listen for new peerconnections, but will signal the active BGP4 to show its presencethrough the TCP/IP network linking the active and the backup.

FIG. 1 is a block diagram illustrating a hardware overview of a HA-BGP4system, in accordance with embodiments of the present invention. RouterA10 contains two separate control plane hardware processors, indicated byACTIVE-BGP4 11 and BACKUP-BGP4 12. ACTIVE-BGP4 11 runs the active BGP4protocol software, and BACKUP-BGP4 12 runs the backup BGP4 protocolsoftware. Link 13 between ACTIVE-BGP4 11 and BACKUP-BGP4 12 is a networkthat supports TCP/IP protocols. One example of such a network isethernet. RouterB 14, RouterC 15 and RouterD 16 are BGP peer routersthat have established BGP connections with RouterA 10.

ACTIVE-BGP4 11 will be brought up to establish BGP connections with BGPpeers, i.e. RouterB 14, RouterC 15 and RouterD 16. When the connectionsare established, ACTIVE-BGP4 11 and BGP peers 14, 15, 16 exchange prefixrouting information. This is usually called exchange of BGP routingtables between the peers. ACTIVE-BGP4 11 selects its best routes amongall of its received and self-configured routes. A basic best routeselection process is defined in RFC 1771. Although most modern BGP4implementations use more complicated rules for selecting best routes,these enhancements are not relevant to high availability BGP4.ACTIVE-BGP4 11 advertises only its best routes to BGP peers 14, 15, 16.

BACKUP-BGP4 12 can be brought up at any time after ACTIVE-BGP4 11 is up.BACKUP-BGP4 12 signals ACTIVE-BGP4 11 to indicate its existence. Thiscan be accomplished by, but is not limited to, establishing a TCP/IPconnection, for example link 13, between ACTIVE-BGP4 11 and BACKUP-BGP412. Active BGP4 instance 11 begins a synchronization process with backupBGP4 instance 12 by copying its running configuration to backup BGP4 12.Then for each established BGP4 connection to a peer router 14, 15, 16,active BGP4 instance 11 transmits the routes learned from that peerrouter to backup BGP4 instance 12. Backup BGP4 instance 12 processes theroutes from each peer router 14-16 just as they were learned by thepeer, except that backup BGP4 instance 12 does not advertise anything toany peers.

Active BGP4 instance 11 then clones onto backup BGP4 instance 12 a TCPsocket that represents each peer connection. This socket cloningoperation is supported by operating system capabilities (see U.S.application Ser. No. 09/852,223, cited above, the disclosure of whichhas been incorporated herein by reference). When socket cloning iscomplete, active BGP4 11 and backup BGP4 12 can start reading from thecloned sockets to learn routes from peer routers 14-16. Only active BGP4instance 11 advertises new routes to the peer, whereas backup BGP4 12does not advertise any routes. All new connection operations, closeoperations, notification operations, new configuration changes, and thelike are handled by active BGP4 instance 11 and are reflected throughthe cloned sockets onto backup BGP4 instance 12.

Then ACTIVE-BGP4 11 and BACKUP-BGP4 12 perform operation to establishrunning state synchronization. ACTIVE-BGP4 11 sends its runningconfiguration to BACKUP-BGP4 12 through link 13.

For each connected BGP peer, for example RouterB 14, ACTIVE-BGP4 11sends the routes received from RouterB 14 to BACKUP-BGP4 12, whichprocesses these routes just as if sent directly from RouterB. FIG. 2 isa block diagram illustrating routing information flow in a HA-BGP4protocol environment. SocketB 21 represents the socket that connectsRouterA 10 to RouterB 14. The socket that represents TCP/IP connection22 between ACTIVE-BGP4 11 and RouterB 14 is cloned onto BACKUP-BGP4 12as cloneB 23. This clone operation is supported on the operating system,for example CHIAROS (see U.S. application Ser. No. 09/852,223, citedabove, the disclosure of which has been incorporated herein byreference). After this clone operation is successful, TCP data 22 sentfrom RouterB 14 is received by both ACTIVE-BGP4 11 and BACKUP-BGP4 12.However, only ACTIVE-BGP4 11 performs route update, listens for new BGPpeer connection requests, and performs new connections. RouterB 14 isthen marked SYNCHRONIZED. If there exist other connection-establishedpeers, for example, RouterC 15 and RouterD 16, that are not markedSYNCHRONIZED, then the operations described above in connection withFIG. 2 are repeated for each such peer.

FIG. 3 is a block diagram representing the state of HA instancesACTIVE-BGP4 11 and BACKUP-BGP4 12 during fail-over and recovery. At step301, after all connected BGP peers 14, 15, 16 are marked SYNCHRONIZED,as described above in connection with FIG. 2, ACTIVE-BGP4 11 is markedACTIVE-PROTECTED and BACKUP-BGP4 12 is marked BACKUP-PROTECT. Whenactive BGP4 instance 11 has copied all of its route database to backupBGP4 instance 12 and has cloned all its established peer connectionsockets, it enters an ACTIVE-PROTECTED state. At the same time, backupBGP4 instance 12 enters a BACKUP-PROTECT state. After this state isreached, if ACTIVE-BGP4 11 fails at step 302, then at step 303BACKUP-BGP4 12 transitions to become new ACTIVE-BGP4 12, and will startto listen for new connections and to send updates, while failedACTIVE-BGP4 remains offline.

Alternatively, if BACKUP-BGP4 12 fails in theACTIVE-PROTECTED/BACKUP-PROTECT state as at step 304, then ACTIVE-BGP411 continues at step 305 as active BGP instance without losing any BGPpeer connections, while failed BACKUP-BGP4 12 remains offline. Duringany phase described above, if backup BGP4 instance 12 fails, active BGP4instance 11 is not affected, and BGP4 peer routers 14-16 will not detectthat anything has happened.

After BGP4 instances 11, 12 enter ACTIVE-PROTECTED/BACKUP-PROTECTstates, if a hardware/software failure, for example power failure,software abnormal exit, or operator-forced failover or offline operationoccurs on active BGP4 instance 11, backup BGP4 instance 12 cooperativelywith the OS detects the failure on active BGP4 instance 11 within onesecond, and backup BGP4 instance 12 transitions itself to become newactive BGP4 instance 12. While backup BGP4 12 transitions itself tobecome active BGP4 12, all existing established peer connections arekept intact. The connected peers will remain unaware that anythingdifferent has happened. After backup BGP4 12 transitions itself to newactive BGP4 12, it starts to listen for new connections from peerrouters and to advertise routes, as needed.

When failed active BGP4 instance 11 is eventually repaired or upgradedand operable, for example with new hardware or new software or both, itcan be brought up again to become new backup BGP4 instance 11, and acomplete life-cycle starts again.

The benefit of this method is that RouterA 10 can tolerate a singlepoint failure such that its BGP peers 14-16 cannot detect the failure.More stable Internet routing can thereby be achieved and maintained.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

What is claimed is:
 1. A method, comprising: maintaining a first routing-protocol instance in a first control unit in a router; maintaining a second routing-protocol instance in a second control unit in the router, wherein the first control unit and second control unit are on separate hardware; maintaining a first transmission-layer socket in the first routing-protocol instance for communication between the router and a peer router; cloning onto the second routing-protocol instance a second transmission-layer socket based on the first transmission-layer socket; and simultaneously receiving route-related data from the peer router at the first and second routing-protocol instances via the first and second sockets, respectively.
 2. The method of claim 1, wherein the routing protocol is Border Gateway Protocol (BGP).
 3. The method of claim 1, further comprising synchronizing the configuration between the first routing-protocol instance and the second routing-protocol instance.
 4. The method of claim 1, wherein the first routing-protocol instance and second routing-protocol instance are maintained by separate operating system software.
 5. The method of claim 1, wherein the first routing-protocol instance is an active instance and the second routing-protocol instance is a backup instance; and wherein in response to a failure in the router, the method further comprises activating the second routing-protocol instance without the failure being detected by the peer router.
 6. A routing system, comprising: a processor; a memory; a first control unit maintains a first routing-protocol instance and a first transmission-layer socket in the first routing-protocol instance for communication between the router and a peer router; and a second control unit maintains a second routing-protocol instance, wherein the first control unit and second control unit are on separate hardware; wherein the second routing-protocol instance comprises a second transmission-layer socket that is cloned based on the first transmission-layer socket; and wherein the first and second routing-protocol instances simultaneously receive route-related data from the peer router via the first and second sockets, respectively.
 7. The routing system of claim 6, wherein the routing protocol is Border Gateway Protocol (BGP).
 8. The routing system of claim 6, wherein the configuration between the first routing-protocol instance and the second routing-protocol instance is synchronized.
 9. The routing system of claim 6, wherein the first routing-protocol instance and second routing-protocol instance are maintained by separate operating system software.
 10. The routing system of claim 6, wherein the first routing-protocol instance is an active instance and the second routing-protocol instance is a backup instance; and wherein in response to a failure in the router, the second routing-protocol instance is activated without the failure being detected by the peer router.
 11. A routing means, comprising: a first control means for maintaining a first routing-protocol instance and a first transmission-layer socket means for communication between the routing means and peer router; and a second control means for maintaining a second routing-protocol instance, wherein the first control means and second control means are on separate hardware; wherein the second routing-protocol instance comprises a second transmission-layer socket means that is cloned based on the first transmission-socket means; and wherein the first and second routing-protocol instances simultaneously receive route-related data from the peer router via the first and second socket means, respectively.
 12. The routing means of claim 11, wherein the routing protocol is Border Gateway Protocol (BGP).
 13. The routing means of claim 11, wherein the configuration between the first control means and the second control means is synchronized.
 14. The routing means of claim 11, wherein the first control means and second control means are maintained by separate operating system software.
 15. The routing means of claim 11, wherein the first control means is an active means and the second control means is a backup means; and wherein in response to a failure in the routing means, the second control means is activated without the failure being detected by the peer router. 